This document analyses the self-developed scheduler component in Android-based mobile devices as part of the SyMptOMS project. This sofware component tested here is called NativeScript Task Dispatcher, that’s a NativeScript plugin aimed to ease mobile app’s tasks definition and the execution of task workflows in background, regardless of whether the app is visible to the user or not. NativeScript Task Dispatcher is publicy available as open source software.

Reproduce notebook

To create the PDF of the computational notebook you can run the following commands in a new R session. If you have problems rendering the PDF you can execute each chunk independently in RStudio.

This document does not install the required R packages by default. You can run the script install.R to install all required dependencies on a new R installation, or use install.packages(..) to install missing R packages.

The plots and tables use the packages ggplot2, knitr::kable(), kableExtra and timetk.

Required libraries and runtime environment description are as follows.

library(tidyverse)
library(kableExtra)
library(here)
# library(googledrive)
library(lubridate)
library(stringr)
library(scales)
# library(plotly)
library(grid)
library(gridBase)
library(gridExtra)
# library(egg)
# library(timetk)

Data download

Log files are stored in shared gdrvie folder. This notebook downloads them locally in the folder data-raw.

Table. List of raw data files.
experiment gsheets_name gsheets_link
#1 AD_BQ.csv https://drive.google.com/open?id=1fP9r0S8ORa689yHRtfCtYQothLuGcCw3
#1 AD_NV.csv https://drive.google.com/open?id=1l-vnaT-Smy0SegArIz5A108-IQ2_jmVV
#1 AD_A1.csv https://drive.google.com/open?id=1gzk1ezN5t5yBv2RnbznUqvalDx97eu3o
#1 BA_H9.csv https://drive.google.com/open?id=1BLXO4Rvz6ppWFJe5oN9Y0hKh__ypofxk
#1 BA_MO.csv https://drive.google.com/open?id=1RizGjKR8QLOdJna7qGPP7dYX9WWAysO4
#2 BA_BQ.csv https://drive.google.com/open?id=1rfaooLZ0Up0gNx1SCA92TV1JFxcLU0sk
#2 BA_NV.csv https://drive.google.com/open?id=1_KIXX5FGXv7MO37QbUDnd_KGqiGihiMI
#2 BA_A1.csv https://drive.google.com/open?id=1LezrtyB9i4K7cQsmI78E0v_KaOyhZcXI
#2 AD_H9.csv https://drive.google.com/open?id=1ISjsfSLZt6n9miryQtDr-tsZV9bfbCDT
#2 AD_MO.csv https://drive.google.com/open?id=1bx2RVNz05qVKGr-32GI-ZIOKQWlqKBPz

Data preparation

Raw data files are cleaned and merged together into a single file stored in the folder data. New fields such as dates breakdown are computed too.

Key Variables:

Exploratory analysis

Summary statistics

Table. Descriptive statistics of data distribution and centrality per scheduler/device.
# exp Type scheduler Device # records Start End Min Bottom whisker 1st Qu. Median Mean 3rd Qu. Top whisker Max
#1 Advanced BQ Aquaris V 21391 2020-03-30 12:15:03 2020-04-14 13:07:57 -48.591 0.235 0.546 0.651 0.769 0.754 1.065 61.204
Nvidia Shield Tablet 21453 2020-03-30 12:15:03 2020-04-14 13:08:14 -52.811 -0.882 0.052 0.241 0.673 1.043 2.518 103.849
Xiaomi Mi A1 21465 2020-03-30 12:23:27 2020-04-14 13:07:50 -60.925 -0.840 0.088 0.269 0.493 0.838 1.945 68.872
Basic Honor 9 21574 2020-03-30 12:00:03 2020-04-14 13:01:35 -0.824 -0.094 0.068 0.087 0.311 0.187 0.364 295.963
Motorola Moto G 21619 2020-03-30 12:00:07 2020-04-14 13:01:16 -0.183 -0.183 0.062 0.100 0.875 0.289 0.629 60.896
#2 Advanced Honor 9 19377 2020-04-15 11:00:02 2020-04-29 12:59:15 -59.535 -0.423 0.167 0.225 0.591 0.842 1.853 93.032
Motorola Moto G 19735 2020-04-16 12:09:53 2020-04-30 12:58:44 -30.166 -0.212 0.117 0.167 2.092 0.782 1.778 126.487
Basic BQ Aquaris V 20146 2020-04-15 11:16:26 2020-04-29 13:00:20 -0.111 0.065 0.280 0.297 0.390 0.424 0.639 34.412
Nvidia Shield Tablet 3528 2020-04-15 11:16:18 2020-04-17 22:27:06 -0.871 -0.871 0.021 0.122 0.431 1.007 2.085 4.841
Xiaomi Mi A1 10832 2020-04-15 11:16:23 2020-04-23 00:52:00 -0.905 -0.661 0.066 0.302 0.405 0.614 1.416 108.035

Observations on the above table :

  • 181.120 records in total.

  • 50.6% (91.582/181.120) is the interquartile range or IQR, i.e. the middle 50% of the data.

  • 39.6% (71.769/181.120) represent the reasonable extremes of the data, i.e. these whisker values are the minimum and maximum values that do not exceed 1.5 × IQR.

  • 9.8% (17.769/181.120) are outliers, i.e. values beyond the top and bottom whiskers.

  • Nvidia Shield Tablet has stopped triggering alarms on 2020-04-17 22:27:06 when running the basic scheduler. Since there’s no watchdog in the basic version, this device is not going to trigger alarms anymore. Died!

  • Xiaomi Mi A1 has stopped triggering alarms on 2020-04-23 00:52:00 when running the basic scheduler. An explanation may be that Huawei/Honor smartphones on Android 9+ have a Launch Records where the OS records how much times an app has been launched by itself, so the OS has the possibility to detect this kind of apps. In these smartphones, Huawei also introduced a new task killer (Power Genius) which kills everything that is not whitelisted by Huawei, and does not give users any configuration options. Therefore, the Power Genius uses the information from the Launch Records to kill the scheduled alarms

Q1: Which one of the analysed schedulers is more reliable? (Distribution of outliers)

In mental health interventions, especially in EMI, the predictibility of the execution of planned actions is mostly preferred since it means notifications and/or alerts are delivered to patients at the right moment. It is inconceivable, for example, that a patient gets a notification (e.g. action, alert, psyco content, etc.) 2 minutes later than the event (e.g. passing nearby a gambling hall) that triggered the action; the patient could be dozens of meters away from the place of interest or, even worse, walked into that no-desirable place. Therefore, closing both the temporal and spatial gaps between the triggered event and the response to the event is critical in mental health EMIs to ensure that time and space/place are in synchrony with respect to the action. If action-time-place(space) are not in synchrony, the intervetion becomes far less effective. Predictibiliy is then a necessary (but not sufficient) proxy for effectiveness.

Given that average response time (delay) is similar in all devices (< 1 second), outliers are a good candidate to study reliability/predictibility. In other words, if a scheduler shows fewer outliers (in terms of delay), it would mean that planned actions are more stable and predictable over time/space and, consequently, that scheduler does a good job in term of effectiveness. On the other hand, a high number of and/or frequent outliers would signal lack of synchrony between space/place and time dimensions in an intervention, resulting in a less predictable scheduler for EMI/A.

Table. Outliers per scheduler/device.
Device # exp Type scheduler # outliers # outliers positive # outliers negative
BQ Aquaris V #1 Advanced 1646 925 721
BQ Aquaris V #2 Basic 2996 2499 497
Honor 9 #2 Advanced 307 209 98
Honor 9 #1 Basic 5033 5018 15
Motorola Moto G #2 Advanced 2861 2852 9
Motorola Moto G #1 Basic 4377 4377 NA
Nvidia Shield Tablet #1 Advanced 294 285 9
Nvidia Shield Tablet #2 Basic 3 3 NA
Xiaomi Mi A1 #1 Advanced 224 121 103
Xiaomi Mi A1 #2 Basic 28 27 1
scheduler n_total
Advanced 5332
Basic 12437

BQ Aquaris V

  • Number of outliers: 4642

  • Number of outliers (top range, beyond top whisker): 3424 - 73.8% (3.424/4.642).

  • Number of outliers (bottom range, beyond bottom whisker): 1218 - 26.2% (1.218/4.642).

Nvidia Shield Tablet

  • Number of outliers: 297

  • Number of outliers (top range, beyond top whisker): 288 - 97% (288/297).

  • Number of outliers (bottom range, beyond bottom whisker): 9 - 3% (9/297).

Xiaomi Mi A1

  • Number of outliers: 252

  • Number of outliers (top range, beyond top whisker): 148 - 58.7% (148/252).

  • Number of outliers (bottom range, beyond bottom whisker): 104 - 41.3% (104/252).

Honor 9

  • Number of outliers: 5340

  • Number of outliers (top range, beyond top whisker): 5227 - 97.9% (5.227/5.340).

  • Number of outliers (bottom range, beyond bottom whisker): 113 - 2.1% (113/5.340).

Motorola Moto G

  • Number of outliers: 7238

  • Number of outliers (top range, beyond top whisker): 7229 - 99.9% (7.229/7.238).

  • Number of outliers (bottom range, beyond bottom whisker): 9 - 0.1% (9/7.238).

Advanced scheduler vs basic scheduler

Centrality statistics of outliers

scheduler device_name mean median
Advanced BQ Aquaris V 2.1539143 1.0830
Advanced Honor 9 9.5330423 3.2580
Advanced Motorola Moto G 12.7074159 9.0550
Advanced Nvidia Shield Tablet 13.6128095 10.9860
Advanced Xiaomi Mi A1 5.0634241 8.5705
Basic BQ Aquaris V 0.7612009 0.9000
Basic Honor 9 1.0521444 1.0180
Basic Motorola Moto G 3.8990078 1.5000
Basic Nvidia Shield Tablet 3.4696667 2.8620
Basic Xiaomi Mi A1 11.1098214 1.6820
scheduler mean median
Advanced 8.995551 4.9965
Basic 2.007192 1.0030

Concluding remarks (Q1)

In response to the question Which one of the two schedulers is more reliable?, we observe the following statements based on the metric delay.

Feasibility on the execution of schedulers:

  • Basic scheduler stopped running in 2 (out of 5) devices.

  • Advanced scheduler completed the experiments on all devices (Note: to clarify issue with Honor 9)

Number of outliers:

  • The number of outliers in the experiments with the basic scheduler is more than double that of the experiments with the advanced scheduler. Therefore,the advanced scheduler beats the basic one in terms of number of outliers.

  • Honor 9 shows a remarkly difference between the two schedulers. This is importnat because this device was used regularly while runing the experiments, demonstrated by the distinct distribution of outliers at nigthtime and daytime.

  • Majority of outliers are positive (beyond top whisker). Negative values (beyond bottom whisker) should be interpreted as wrong values due to planned timestamps mistakenly recorded during data collection.

Frequency of outliers:

  • It is not observed a temporal pattern regarding the frequency of outliers for devices used exclusively for the experiments. Outliers are generally evenly distributed over time.

Centrality measurements of outliers:

  • Basic scheduler shows delays with greater centrality measurements than advanced scheduler.

Q2: Which one of the analysed schedulers is more accurate?

We assume that higher accuracy equals to lower delay. In Q1, we compute delayas execution timestamp(i) - planned timestamp(i), being i each time step in the serie (roughly 1 minute). Here, we compute other types of delays to make sure that the scheduler works as expected. Therefore, we compute the following timestamps differences:

  • Consecutive planning timestamps, planned timestamp(i-1) - planned timestamp(i), in order to detect wrong planned timestamp values

  • Consecutive execution timestamps, execution timestamp(i-1) - execution timestamp(i), to detect execution timestamps between consecutive measurements

  • Overhead, execution timestamp(i) - planning timestamp(i+1), to explore framework overhead between consecutive measurements

Consecutive planning timestamps (delay1)

We calculate planned timestamp(i-1) - planned timestamp(i), to detect wrong planning timestamp values. During data collection, chances are that some planned timestamps are not correctly registered, creating duplicated timestamps, suggested by the presence of negative values of delay. The new variable/metric is called delay1.

Table. Descriptive statistics of delay1 (planned timestamp(i-1) - planned timestamp(i)) per scheduler/device.
# exp Type scheduler Device # records Start End Min Bottom whisker 1st Qu. Median Mean 3rd Qu. Top whisker Max
#1 Advanced BQ Aquaris V 21391 2020-03-30 12:15:03 2020-04-14 13:07:57 -60 0.159 0.463 0.565 0.735 0.666 0.970 1263.965
Nvidia Shield Tablet 21453 2020-03-30 12:15:03 2020-04-14 13:08:14 -60 -1.371 0.012 0.179 0.560 0.980 2.415 131.099
Xiaomi Mi A1 21465 2020-03-30 12:23:27 2020-04-14 13:07:50 -60 -1.064 0.039 0.224 0.501 0.796 1.887 568.508
Basic Honor 9 21574 2020-03-30 12:00:03 2020-04-14 13:01:35 -60 -0.093 0.014 0.025 0.243 0.103 0.235 236.462
Motorola Moto G 21619 2020-03-30 12:00:07 2020-04-14 13:01:16 -60 -0.020 0.008 0.016 0.117 0.027 0.055 60.719
#2 Advanced Honor 9 19377 2020-04-15 11:00:02 2020-04-29 12:59:15 -60 -0.574 0.063 0.109 2.794 0.738 1.748 44875.248
Motorola Moto G 19735 2020-04-16 12:09:53 2020-04-30 12:58:44 -60 -0.267 0.032 0.064 1.441 0.378 0.895 126.625
Basic BQ Aquaris V 20146 2020-04-15 11:16:26 2020-04-29 13:00:20 -60 0.045 0.247 0.262 0.351 0.385 0.591 34.369
Nvidia Shield Tablet 3528 2020-04-15 11:16:18 2020-04-17 22:27:06 -60 -0.887 0.002 0.098 0.388 0.979 2.029 4.777
Xiaomi Mi A1 10832 2020-04-15 11:16:23 2020-04-23 00:52:00 -60 -0.715 0.009 0.269 0.352 0.579 1.429 108.009
Table. Outliers of delay1 per scheduler/device.
Device Type scheduler # outliers # outliers positive # outliers negative
BQ Aquaris V Advanced 1702 888 814
BQ Aquaris V Basic 3093 2525 568
Honor 9 Advanced 455 263 192
Honor 9 Basic 5067 5048 19
Motorola Moto G Advanced 2593 2563 30
Motorola Moto G Basic 3659 3287 372
Nvidia Shield Tablet Advanced 198 181 17
Nvidia Shield Tablet Basic 6 4 2
Xiaomi Mi A1 Advanced 360 181 179
Xiaomi Mi A1 Basic 23 20 3

Consecutive execution timestamps (delay2)

We calculate execution timestamp(i-1) - execution timestamp(i) which gives us whether execution times between consecutive measurements (i.e. tasks execution) are accurate, i.e., tasks are executed as expected. The new variable/metric is called delay2 (substracting 60 to get values around zero).

Table. Descriptive statistics of ‘delay2’ (execution timestamp(n-1) - execution timestamp(n)) per scheduler/device.
# exp Type scheduler Device # records Start End Min Bottom whisker 1st Qu. Median Mean 3rd Qu. Top whisker Max
#1 Advanced BQ Aquaris V 21391 2020-03-30 12:15:03 2020-04-14 13:07:57 -60 0.157 0.462 0.565 0.737 0.666 0.971 1264.016
Nvidia Shield Tablet 21453 2020-03-30 12:15:03 2020-04-14 13:08:14 -60 -1.277 0.002 0.182 0.562 0.982 2.392 56.995
Xiaomi Mi A1 21465 2020-03-30 12:23:27 2020-04-14 13:07:50 -60 -0.877 0.042 0.231 0.501 0.803 1.888 568.652
Basic Honor 9 21574 2020-03-30 12:00:03 2020-04-14 13:01:35 -60 -0.157 0.010 0.028 0.243 0.122 0.290 295.912
Motorola Moto G 21619 2020-03-30 12:00:07 2020-04-14 13:01:16 -60 -0.256 -0.024 0.023 0.118 0.131 0.363 60.747
#2 Advanced Honor 9 19377 2020-04-15 11:00:02 2020-04-29 12:59:15 -60 -0.910 0.062 0.118 2.794 0.744 1.758 44875.732
Motorola Moto G 19735 2020-04-16 12:09:53 2020-04-30 12:58:44 -60 -0.899 0.024 0.080 1.441 0.648 1.573 87.974
Basic BQ Aquaris V 20146 2020-04-15 11:16:26 2020-04-29 13:00:20 -60 0.043 0.247 0.262 0.351 0.383 0.587 34.369
Nvidia Shield Tablet 3528 2020-04-15 11:16:18 2020-04-17 22:27:06 -60 -0.887 -0.003 0.099 0.388 0.979 2.062 4.704
Xiaomi Mi A1 10832 2020-04-15 11:16:23 2020-04-23 00:52:00 -60 -0.695 0.021 0.269 0.352 0.578 1.402 108.004
Table. Outliers of delay2 per scheduler/device.
Device Type scheduler # outliers # outliers positive # outliers negative
BQ Aquaris V Advanced 19408 19367 41
BQ Aquaris V Basic 5292 5286 6
Honor 9 Advanced 6603 6560 43
Honor 9 Basic 5011 4990 21
Motorola Moto G Advanced 7348 5937 1411
Motorola Moto G Basic 7046 4508 2538
Nvidia Shield Tablet Advanced 10520 9857 663
Nvidia Shield Tablet Basic 1600 1516 84
Xiaomi Mi A1 Advanced 9373 9336 37
Xiaomi Mi A1 Basic 4082 4034 48

delay2 shows strange values only for the Motorola Moto G. Below we explore its distribution per each day: delay2 shows abnormal values during the second week (from 25 to 30) of the advance scheduler (red).

Overhead between consecutive measurements

Concluding remarks (Q2)

Q3: Does the battery level influence timely alarm execution? (Distribution of delay over battery - without outliers)

Concluding remarks (Q3)

Q4: Which one of the analysed schedulers loses fewer alarm executions?

Higher reliability = higher percentage of alarms executed / expected alarms for the experiment time frame

Concluding remarks (Q4)

Summary

DISCARD TEXT/CODE BEYOND THIS LINE

Boxplots

Distribution of data based on a five number summary (“minimum”, first quartile (Q1), median, third quartile (Q3), and “maximum”).

Basic statistics applied to delay and device_name. Given a device, how do the box plots change on each scheduler?